Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium

We develop provably efficient reinforcement learning algorithms for two-player zero-sum finite-horizon Markov games with simultaneous moves. To incorporate function approximation, we consider a family of where the reward and transition kernel possess linear structure. Both offline online settings problems are considered. In setting, control both players aim to find Nash equilibrium by minimizing duality gap. single player playing against an arbitrary opponent minimize regret. For settings, propose optimistic variant least-squares minimax value iteration algorithm. show that our algorithm is computationally achieves [Formula: see text] upper bound on gap regret, d dimension, H horizon T total number timesteps. Our results do not require additional assumptions sampling model. setting requires overcoming several new challenges absent in decision processes or turn-based games. particular, achieve optimism moves, construct lower confidence bounds function, then compute policy solving general-sum matrix game these as payoff matrices. As finding hard, instead solves coarse correlated (CCE), which can be obtained efficiently. best knowledge, such CCE-based scheme has appeared literature might interest its own right. Funding: Q. Xie partially supported National Science Foundation [Grant CNS-1955997] J.P. Morgan. Y. Chen [Grants CCF-1657420, CCF-1704828, CCF-2047910]. Z. Wang acknowledges 2048075, 2008827, 2015568, 1934931], Simons Institute (Theory Reinforcement Learning), Amazon, Morgan, Two Sigma their support.